Simulation Of Spiking Neural Networks Research Articles

Scaling neural simulations in STACS

As modern neuroscience tools acquire more details about the brain, the need to move towards biological-scale neural simulations continues to grow. However, effective simulations at scale remain a challenge. Beyond just the tooling required to enable parallel execution, there is also the unique structure of the synaptic interconnectivity, which is globally sparse but has relatively high connection density and non-local interactions per neuron. There are also various practicalities to consider in high performance computing applications, such as the need for serializing neural networks to support potentially long-running simulations that require checkpoint-restart. Although acceleration on neuromorphic hardware is also a possibility, development in this space can be difficult as hardware support tends to vary between platforms and software support for larger scale models also tends to be limited.In this paper, we focus our attention on Simulation Tool for Asynchronous Cortical Streams (STACS), a spiking neural network simulator that leverages the Charm++ parallel programming framework, with the goal of supporting biological-scale simulations as well as interoperability between platforms. Central to these goals is the implementation of scalable data structures suitable for efficiently distributing a network across parallel partitions. Here, we discuss a straightforward extension of a parallel data format with a history of use in graph partitioners, which also serves as a portable intermediate representation for different neuromorphic backends.We perform scaling studies on the Summit supercomputer, examining the capabilities of STACS in terms of network build and storage, partitioning, and execution. We highlight how a suitably partitioned, spatially dependent synaptic structure introduces a communication workload well-suited to the multicast communication supported by Charm++. We evaluate the strong and weak scaling behavior for networks on the order of millions of neurons and billions of synapses, and show that STACS achieves competitive levels of parallel efficiency.

Release 2.0 — NEMSIM-RT: A real-time distributed spiking neural network simulator

NESIM-RT is a specialized tool designed for simulating neuromorphic systems. In this new release we extend its capabilities to include state-of-the art models like the AdexLIF and Izhikevich, and to incorporate dynamic synaptic mechanisms such as Spike-Timing Dependent Plasticity (STDP). With these new features, researchers can now observe in real-time how different parameters influence these models and learning rules, thereby gaining deeper insights into neuronal function and network dynamics.

Accelerating spiking neural network simulations with PymoNNto and PymoNNtorch.

Spiking neural network simulations are a central tool in Computational Neuroscience, Artificial Intelligence, and Neuromorphic Engineering research. A broad range of simulators and software frameworks for such simulations exist with different target application areas. Among these, PymoNNto is a recent Python-based toolbox for spiking neural network simulations that emphasizes the embedding of custom code in a modular and flexible way. While PymoNNto already supports GPU implementations, its backend relies on NumPy operations. Here we introduce PymoNNtorch, which is natively implemented with PyTorch while retaining PymoNNto's modular design. Furthermore, we demonstrate how changes to the implementations of common network operations in combination with PymoNNtorch's native GPU support can offer speed-up over conventional simulators like NEST, ANNarchy, and Brian 2 in certain situations. Overall, we show how PymoNNto's modular and flexible design in combination with PymoNNtorch's GPU acceleration and optimized indexing operations facilitate research and development of spiking neural networks in the Python programming language.

Hippocampome.org 2.0 is a knowledge base enabling data-driven spiking neural network simulations of rodent hippocampal circuits.

Hippocampome.org is a mature open-access knowledge base of the rodent hippocampal formation focusing on neuron types and their properties. Previously, Hippocampome.org v1.0 established a foundational classification system identifying 122 hippocampal neuron types based on their axonal and dendritic morphologies, main neurotransmitter, membrane biophysics, and molecular expression (Wheeler et al., 2015). Releases v1.1 through v1.12 furthered the aggregation of literature-mined data, including among others neuron counts, spiking patterns, synaptic physiology, in vivo firing phases, and connection probabilities. Those additional properties increased the online information content of this public resource over 100-fold, enabling numerous independent discoveries by the scientific community. Hippocampome.org v2.0, introduced here, besides incorporating over 50 new neuron types, now recenters its focus on extending the functionality to build real-scale, biologically detailed, data-driven computational simulations. In all cases, the freely downloadable model parameters are directly linked to the specific peer-reviewed empirical evidence from which they were derived. Possible research applications include quantitative, multiscale analyses of circuit connectivity and spiking neural network simulations of activity dynamics. These advances can help generate precise, experimentally testable hypotheses and shed light on the neural mechanisms underlying associative memory and spatial navigation.

Hippocampome.org 2.0 is a knowledge base enabling data-driven spiking neural network simulations of rodent hippocampal circuits

Hippocampome.org is a mature open-access knowledge base of the rodent hippocampal formation focusing on neuron types and their properties. Previously, Hippocampome.org v1.0 established a foundational classification system identifying 122 hippocampal neuron types based on their axonal and dendritic morphologies, main neurotransmitter, membrane biophysics, and molecular expression (Wheeler et al., 2015). Releases v1.1 through v1.12 furthered the aggregation of literature-mined data, including among others neuron counts, spiking patterns, synaptic physiology, in vivo firing phases, and connection probabilities. Those additional properties increased the online information content of this public resource over 100-fold, enabling numerous independent discoveries by the scientific community. Hippocampome.org v2.0, introduced here, besides incorporating over 50 new neuron types, now recenters its focus on extending the functionality to build real-scale, biologically detailed, data-driven computational simulations. In all cases, the freely downloadable model parameters are directly linked to the specific peer-reviewed empirical evidence from which they were derived. Possible research applications include quantitative, multiscale analyses of circuit connectivity and spiking neural network simulations of activity dynamics. These advances can help generate precise, experimentally testable hypotheses and shed light on the neural mechanisms underlying associative memory and spatial navigation.

Review of common spiking neural network simulation tools

As a recognized third-generation neural network, the spiking neural network has high concurrency and complexity. Although the degree of research is still far from the previous generation of neural networks, spiking neural networks are excellent in performance and energy consumption. In this paper, the common spiking neural network simulation tools are reviewed. The most frequently used and mentioned tools are NEURON, NEST, and BRAIN. NEURON is more suitable for simulation based on biological applications, pays more attention to biological characteristics, and can support large-scale network simulation. Examples used in the official documentation are neural simulations of invertebrates and mammals. Large heterogeneous networks of point neurons or neurons with a few compartments are frequently simulated using NEST. In contrast to models that concentrate on the specific morphological and biophysical characteristics of individual neurons, NEST is appropriate for those that emphasize the dynamics, size, and structure of the nervous system. Brian was originally designed for research and teaching and is well suited as a teaching and presentation tool for simulating and observing the effects of different parameters for classical neural network projects such as picture classification. In addition, CSIM, SPLIT, SPINNAKER, and other tools also have their merits, but due to the low frequency of relevant references and lack of universality, this study will not give a detailed introduction.

SHIP: a computational framework for simulating and validating novel technologies in hardware spiking neural networks.

Investigations in the field of spiking neural networks (SNNs) encompass diverse, yet overlapping, scientific disciplines. Examples range from purely neuroscientific investigations, researches on computational aspects of neuroscience, or applicative-oriented studies aiming to improve SNNs performance or to develop artificial hardware counterparts. However, the simulation of SNNs is a complex task that can not be adequately addressed with a single platform applicable to all scenarios. The optimization of a simulation environment to meet specific metrics often entails compromises in other aspects. This computational challenge has led to an apparent dichotomy of approaches, with model-driven algorithms dedicated to the detailed simulation of biological networks, and data-driven algorithms designed for efficient processing of large input datasets. Nevertheless, material scientists, device physicists, and neuromorphic engineers who develop new technologies for spiking neuromorphic hardware solutions would find benefit in a simulation environment that borrows aspects from both approaches, thus facilitating modeling, analysis, and training of prospective SNN systems. This manuscript explores the numerical challenges deriving from the simulation of spiking neural networks, and introduces SHIP, Spiking (neural network) Hardware In PyTorch, a numerical tool that supports the investigation and/or validation of materials, devices, small circuit blocks within SNN architectures. SHIP facilitates the algorithmic definition of the models for the components of a network, the monitoring of states and output of the modeled systems, and the training of the synaptic weights of the network, by way of user-defined unsupervised learning rules or supervised training techniques derived from conventional machine learning. SHIP offers a valuable tool for researchers and developers in the field of hardware-based spiking neural networks, enabling efficient simulation and validation of novel technologies.

Chip-In-Loop SNN Proxy Learning: a new method for efficient training of spiking neural networks.

The primary approaches used to train spiking neural networks (SNNs) involve either training artificial neural networks (ANNs) first and then transforming them into SNNs, or directly training SNNs using surrogate gradient techniques. Nevertheless, both of these methods encounter a shared challenge: they rely on frame-based methodologies, where asynchronous events are gathered into synchronous frames for computation. This strays from the authentic asynchronous, event-driven nature of SNNs, resulting in notable performance degradation when deploying the trained models on SNN simulators or hardware chips for real-time asynchronous computation. To eliminate this performance degradation, we propose a hardware-based SNN proxy learning method that is called Chip-In-Loop SNN Proxy Learning (CIL-SPL). This approach effectively eliminates the performance degradation caused by the mismatch between synchronous and asynchronous computations. To demonstrate the effectiveness of our method, we trained models using public datasets such as N-MNIST and tested them on the SNN simulator or hardware chip, comparing our results to those classical training methods.

Circuit implementation of on-chip trainable spiking neural network using CMOS based memristive STDP synapses and LIF neurons

Computation on a large volume of data at high speed and low power requires energy-efficient architectures for edge computing applications. As a result, scientists focus on memristive circuits and systems for area and energy efficiency. Spiking neural network (SNN) with bio-inspired spike-timing-dependent plasticity learning (STDP) is a promising solution for energy-efficient neuromorphic systems than conventional artificial neural network (ANN). Previous works on SNN with STDP learning primarily use memristor macro models, which are software-based and cannot give complete insight into circuit implementation challenges. Some reported works on SNN use memristive devices, which require additional fabrication steps. This article presents a full circuit-level implementation of the SNN system featuring on-chip training and classification using memristive STDP synapse in standard CMOS technology. A new learning rule using the modified STDP is implemented to simplify the weight modification process. It does not involve FPGAs, CPUs, or GPUs to train the neural network. The approach used in this paper to modify the weights does not require any additional combinational or digital circuits attached to the memristive synapse resulting in less consumption of area, energy and time. We demonstrated the complete circuit-level design, implementation and simulation of SNN with on-chip training and pattern classification using 180 nm CMOS technology. A comprehensive comparison of the proposed SNN circuit with the previous related work is also presented. To demonstrate the versatility of the CMOS synapse circuit for application scenarios requiring rate-based learning, we have tuned the pair-based STDP circuit to obtain Bienenstock–Cooper–Munro (BCM) characteristics and applied it to heart rate classification.

Optimization of post-annealing temperature for high-performance synaptic transistors based on In–Ga–Zn–O channel and trap layers

Synaptic transistors using an indium–gallium–zinc oxide (IGZO) charge trap layer (CTL) have great potential for high-performance neuromorphic applications due to their excellent stability and high linearity in conductance modulation. Since the high conductivity of the IGZO CTL enhances charge de-trapping performance, the post-annealing process, which directly affects the conductivity of the film, should be carefully conducted. In this study, we investigated the effect of post-annealing temperature (TPA) on the charge de-trapping performance of synaptic transistors. The synaptic transistor showed superior charge de-trapping performance at a TPA of 250 °C, resulting from the increased conductivity of the CTL. Through various electrical and chemical analysis methods, it was revealed that the hydrogen decrease within the CTL and the oxidation–reduction reaction at the interface between the tunneling layer and the CTL contributed to the significant increase in the conductivity of the CTL. The synaptic transistor with the optimal TPA exhibited high linearity of potentiation (αp: 0.56) and depression (αd: −0.65) with 64 conductance levels, enabling the spiking neural network simulation to achieve a high accuracy of 88.75% using a Fashion-MNIST dataset. This work shows that TPA is one of the critical process strategies for the high charge de-trapping performance of the synaptic transistor by inducing the chemical reaction at the CTL interface and hydrogen content change within the CTL.

Bridging Neuroscience and Robotics: Spiking Neural Networks in Action.

Robots are becoming increasingly sophisticated in the execution of complex tasks. However, an area that requires development is the ability to act in dynamically changing environments. To advance this, developments have turned towards understanding the human brain and applying this to improve robotics. The present study used electroencephalogram (EEG) data recorded from 54 human participants whilst they performed a two-choice task. A build-up of motor activity starting around 400 ms before response onset, also known as the lateralized readiness potential (LRP), was observed. This indicates that actions are not simply binary processes but rather, response-preparation is gradual and occurs in a temporal window that can interact with the environment. In parallel, a robot arm executing a pick-and-place task was developed. The understanding from the EEG data and the robot arm were integrated into the final system, which included cell assemblies (CAs)-a simulated spiking neural network-to inform the robot to place the object left or right. Results showed that the neural data from the robot simulation were largely consistent with the human data. This neurorobotics study provides an example of how to integrate human brain recordings with simulated neural networks in order to drive a robot.

Reconfigurable Mott electronics for homogeneous neuromorphic platform

Abstract To simplify the fabrication process and increase the versatility of neuromorphic systems, the reconfiguration concept has attracted much attention. Here, we developed a novel electrochemical VO2 (EC-VO2) device, which can be reconfigured as synapses or LIF neurons. The ionic dynamic doping contributes to the resistance changes of VO2, which enables the reversible modulation of device states. The analog resistance switching and tunable LIF functions were both measured based on the same device to demonstrate the capacity of reconfiguration. Based on the reconfigurable EC-VO2, the simulated spiking neural network model exhibited excellent performances by using low-precision weights and tunable output neurons, whose final accuracy reached 91.92%.

Fast Simulation of Analog Spiking Neural Network With Device Non-Idealites

We present a method for spiking neural network simulation with hardware realistic device non-idealities present in the neuron and synapse circuits as an alternative to time-consumingexpensive spice simulation. Neuromorphic machine learning algorithms for spiking neural networks are often simulatedtested with an ideal mathematical description of spiking neurons and synapses. However, silicon implementations of spiking neurons differ significantly from their ideal mathematical models because of device non-idealities and restrictions in the range of device operation. These non-idealities affect the performance of chip implementations of spiking networks. The dynamical system phase plane of the neuron and synapse circuit is used to create a compact representation of the neuron dynamics that captures device non-idealities for simulation of spiking neural networks. The proposed method would allow hardware-aware optimization of neuromorphic algorithms using standard machine learning tools and provide simulated network prediction close to what would be expected from a chip implementation.

SWsnn: A Novel Simulator for Spiking Neural Networks.

Spiking neural network (SNN) simulators play an important role in neural system modeling and brain function research. They can help scientists reproduce and explore neuronal activities in brain regions, neuroscience, brain-like computing, and other fields and can also be applied to artificial intelligence, machine learning, and other fields. At present, many simulators using central processing unit (CPU) or graphics processing unit (GPU) have been developed. However, due to the randomness of connections between neurons and spiking events in SNN simulation, this causes a lot of memory access time. To alleviate this problem, we developed an SNN simulator SWsnn based on the new Sunway SW26010pro processor. The SW26010pro processor consists of six core groups, each with 16 MB of local data memory (LDM). LDM has the characteristics of high-speed read and write, which is suitable for performing simulation tasks similar to SNNs. Experimental results show that SWsnn runs faster than other mainstream GPU-based simulators when simulating a certain scale of neural network, showing a strong performance advantage. To conduct larger scale simulations, SWsnn designed a simulation computation based on a large shared model of Sunway processor and developed a multiprocessor version of SWsnn based on this mode, achieving larger scale SNN simulations.

48‐3: In‐Ga‐Zn‐O Synaptic Transistor with 1 µm Channel Length for Neuromorphic Computing

In this paper, indium‐gallium‐zinc‐oxide synaptic transistors with different channel lengths ranging from 2 μm to 1 μm were demonstrated. Channel scaling effect on synaptic behaviors such as long‐term depression and long‐term potentiation was investigated. A spiking neural network simulation was conducted to verify our synaptic transistor with a channel length of 1 μm, achieving high classification accuracy of 98.05%.

Synaptic Transistor Based on In‐Ga‐Zn‐O Channel and Trap Layers with Highly Linear Conductance Modulation for Neuromorphic Computing

AbstractBrain‐inspired neuromorphic computing has drawn significant attraction as a promising technology beyond von Neumann architecture by using the parallel structure of synapses and neurons. Various artificial synapse configurations and materials have been proposed to emulate synaptic behaviors for human brain functions such as memorizing, learning, and visual processing. Especially, the memory type indium‐gallium‐zinc‐oxide (IGZO) synaptic transistor adopting a charge trapping layer (CTL) has the advantages of high stability and a low leakage current of the IGZO channel. However, the CTL material should be carefully selected and optimized to overcome the low de‐trapping efficiency, resulting from difficulty in inducing holes in the IGZO channel. In this paper, IGZO is adopted as a CTL and found out that making it degenerated is crucial to improving de‐trapping efficiency. The degenerate CTL, where electrons remain as free electrons, induces Fowler‐Nordheim tunneling by increasing the electric field across the tunneling layer. As a result, the synaptic transistor represents a high linearity of potentiation (αp: −0.03) and depression (αd: −0.47) with 64 conductance levels, which enables the spiking neural network simulation to achieve high accuracy of 98.08%. These experimental results indicate that the synapse transistor can be one of the promising candidates for neuromorphic applications.

A Novel Structured Single Device Neuron for Low Standby Power and Compact System Application

We propose a single device neuron that utilizes a ferroelectric layer, a split gate, and a truncated floating gate structure. The proposed neuron device, named Single-Device Leaky-FeFET (SD L-FeFET), successfully emulates the neuronal dynamics for both excitatory and inhibitory connections while reducing the standby power by eliminating tail current. A spiking neural network (SNN) using the newly developed SD L-FeFET neuron shows MNIST handwriting digit pattern recognition of 92.5% and face recognition of 91.7%, which is comparable performance to the state-of-art SNN simulation results using conventional complex cell designs.

SPAYK: An environment for spiking neural network simulation

In research areas such as mobile robotics and computer vision, energy and computational efficiency have become critical. This has greatly increased interest in high-efficiency neuromorphic hardware and spiking neural networks. Because neuromorphic hardware is not yet widely available, spiking neural network studies are conducted by simulations. There are numerous simulators available today, each designed for a specific purpose. In this paper, a novel and open source package (SPAYK) for simulating spiking neural networks is presented. SPAYK has been proposed to speed up spiking neural network research. In the majority of simulators, networks are expressed with differential equations and require advanced neuroscience knowledge since such simulators are generally designed for brain and neuroscience research. SPAYK, on the other hand, is specifically designed as a framework to easily design spiking neural networks for practical problems. SPAYK is an easy-to-use Python package. There are three fundamental classes in the core: the model class for creating neuron groups, the organization class for simulating tissues, and the learning class for synaptic plasticity. While developing and testing the SPAYK environment, various experiments were carried out. This study includes three of these experiments. In the first experiment, we investigated the behavior of a group of Izhikevich neurons for visual stimuli. Also, a single Izhikevich neuron has been trained to respond to a particular label in a supervised manner with synaptic plasticity. In the second experiment, a well-known experiment was repeated to validate SPAYK. In this experiment, a neuron trained by synaptic plasticity can recognize repetitive patterns in a spike train. In the third experiment, a similar neuron was simulated with stimuli with multiple labels adapted from the MNIST dataset. It has been shown that the neuron can classify a particular label by synaptic plasticity. All these experiments and the SPAYK environment are presented as open-source tools.

Efficient parameter calibration and real-time simulation of large-scale spiking neural networks with GeNN and NEST.

Spiking neural networks (SNNs) represent the state-of-the-art approach to the biologically realistic modeling of nervous system function. The systematic calibration for multiple free model parameters is necessary to achieve robust network function and demands high computing power and large memory resources. Special requirements arise from closed-loop model simulation in virtual environments and from real-time simulation in robotic application. Here, we compare two complementary approaches to efficient large-scale and real-time SNN simulation. The widely used NEural Simulation Tool (NEST) parallelizes simulation across multiple CPU cores. The GPU-enhanced Neural Network (GeNN) simulator uses the highly parallel GPU-based architecture to gain simulation speed. We quantify fixed and variable simulation costs on single machines with different hardware configurations. As a benchmark model, we use a spiking cortical attractor network with a topology of densely connected excitatory and inhibitory neuron clusters with homogeneous or distributed synaptic time constants and in comparison to the random balanced network. We show that simulation time scales linearly with the simulated biological model time and, for large networks, approximately linearly with the model size as dominated by the number of synaptic connections. Additional fixed costs with GeNN are almost independent of model size, while fixed costs with NEST increase linearly with model size. We demonstrate how GeNN can be used for simulating networks with up to 3.5 · 106 neurons (> 3 · 1012synapses) on a high-end GPU, and up to 250, 000 neurons (25 · 109 synapses) on a low-cost GPU. Real-time simulation was achieved for networks with 100, 000 neurons. Network calibration and parameter grid search can be efficiently achieved using batch processing. We discuss the advantages and disadvantages of both approaches for different use cases.

Brian2CUDA: Flexible and Efficient Simulation of Spiking Neural Network Models on GPUs.

Graphics processing units (GPUs) are widely available and have been used with great success to accelerate scientific computing in the last decade. These advances, however, are often not available to researchers interested in simulating spiking neural networks, but lacking the technical knowledge to write the necessary low-level code. Writing low-level code is not necessary when using the popular Brian simulator, which provides a framework to generate efficient CPU code from high-level model definitions in Python. Here, we present Brian2CUDA, an open-source software that extends the Brian simulator with a GPU backend. Our implementation generates efficient code for the numerical integration of neuronal states and for the propagation of synaptic events on GPUs, making use of their massively parallel arithmetic capabilities. We benchmark the performance improvements of our software for several model types and find that it can accelerate simulations by up to three orders of magnitude compared to Brian's CPU backend. Currently, Brian2CUDA is the only package that supports Brian's full feature set on GPUs, including arbitrary neuron and synapse models, plasticity rules, and heterogeneous delays. When comparing its performance with Brian2GeNN, another GPU-based backend for the Brian simulator with fewer features, we find that Brian2CUDA gives comparable speedups, while being typically slower for small and faster for large networks. By combining the flexibility of the Brian simulator with the simulation speed of GPUs, Brian2CUDA enables researchers to efficiently simulate spiking neural networks with minimal effort and thereby makes the advancements of GPU computing available to a larger audience of neuroscientists.